Comparison of traditional vs novel approach to define cancer-associated bleeding Free

Introduction Bleeding is a frequent yet under-captured complication in cancer patients undergoing systemic therapy. While International Classification of Disease (ICD-10) codes are widely used to identify bleeding, their utility in the research context, particularly in cancer patients, is unclear. The lack of a practical computable phenotype to define cancer-associated bleeding limits both thromboprophylaxis research and implementation science efforts. In this study, we aimed to 1) systematically validate ICD-10 codes for identifying clinically relevant bleeding (CRB) in cancer patients, 2) assess the usability of a natural language processing (NLP) platform (NLPMed) to facilitate physician annotation, and 3) finetune a masked language model (MSM) to detect CRB (Bleed-BERT). Methods To identify candidate ICD-10 codes for CRB, we performed a systematic review of PubMed from 1/2017-4/2024 that reported chart review validation metrics (mostly at time of hospital discharge). Using 6 final studies, we compiled a list of 545 unique codes, which were independently reviewed by two hematologists (JJ and JR) based on clinical and coding relevance to create a final list of 279 bleeding codes. We then applied the ICD codes using the inpatient billing final diagnosis table in 7,460 cancer patients from a single institution to predict bleeding outcomes within 1 year from anti-cancer systemic therapy initiation.

For ICD algorithm validation, we randomly selected 100 patients each from ICD+ and ICD- groups. For gold standard, two trained physicians (JT and EC) independently annotated all clinical notes from these 200 patients using the customized NLPMed portal that contained pre-filtered clinical notes with highlighted bleeding keyword-containing sentences. CRB was defined as either major bleeding (MB) or clinically relevant non-major bleeding (CRNMB) using the International Society on Thrombosis and Haemostasis (ISTH) definitions. Discrepancy was resolved by consensus agreement and hematologist adjudication (AL). We assessed Cohen's kappa (interrater concordance), precision (positive predictive value), and recall (sensitivity).

For NLP algorithm development, we split the 3,514 annotated note labels from 200 patients into train (80%) and test (20%) datasets to finetune Bio_ClinicalBERT for CRB classification. We then tested the model performance on patient-level annotations.

Results From 200 patients, clinician annotation identified 108 CRBs within 1 year, including 71 MB and 50 CRNMB. Several patients had multiple bleeding events. Cohen's Kappa of CRB, MB, and CRNMB was 0.76, 0.72, and 0.58, respectively. Many patients had direct tumor-related or post-procedural bleeds that did not fit well with the ISTH definitions.

Compared to traditional chart review, NLPMed portal allowed for greater efficiency and accuracy for secure annotation without requiring separate log-in to EHR or different recording software (i.e. REDCap). The average time spent was 7 and 1 minute for patients with positive and negative events, respectively. The annotation process generated labeled notes for MB (n=880), CRNMB (n=526), and negative for bleeding (n=2,108).

The precision and recall of the ICD-10 algorithm was 93% and 86%, respectively, for CRB within 1 year (n=200). The same metrics were 63% and 89% for MB and 42% and 84% for CRNMB. In comparison, the precision and recall of Bleed-BERT on the train dataset was 96% and 100%, respectively, for CRB within 1 year (n=160). The same metrics on the test dataset were 96% and 100% (n=40).

Conclusion We demonstrate the validity and limitation of using traditional ICD-10 codes for identifying CRB in cancer patients. We further propose a novel MSM (Bleed-BERT; https://nlpmed.demo.angli-lab.com) to classify bleeding events using unstructured clinical notes. Without accounting for bleeding prevalence, inpatient ICD-10 codes underestimates CRB by 14%. Furthermore, inpatient ICD-10 codes can correlate with either MB (63%) or CRNMB (42%) based on the ISTH definition, though most events (93%) are considered clinically relevant by physician annotators. Unlike the ICD-10 algorithm that relies on professional coder input at hospital discharge, the Bleed-BERT model can interpret clinical notes from both inpatient and ambulatory settings with improved precision and recall. We are actively developing a transformer model to incorporate BERT embeddings with hemoglobin trends and blood transfusions to differentiate bleeding severity.